Enhancing Local Context of Histology Features in Vision Transformers

Predicting complete response to radiotherapy in rectal cancer patients using deep learning approaches from morphological features extracted histology biopsies provides a quick, low-cost and effective way assist clinical decision making. We propose adjustments the Vision Transformer (ViT) network improve utilisation of contextual information present whole slide images (WSIs). Firstly, our position restoration embedding (PRE) preserves spatial relationship between tissue patches, their original positions on WSI. Secondly, clustering analysis explores motifs which capture fundamental biological processes found tumour micro-environment. This is introduced into ViT form cluster label token, helping model differentiate types. The proposed methods are demonstrated two large independent datasets selectively treated with capecitabine UK trials. Experiments demonstrate that both models, PREViT ClusterViT, show improvements prediction over baseline models.